Method of facial image reproduction and related device

ABSTRACT

To modify a facial feature region in a video bitstream, the video bitstream is received and a feature region is extracted from the video bitstream. An audio characteristic, such as frequency, rhythm, or tempo is retrieved from an audio bitstream, and the feature region is modified according to the audio characteristic to generate a modified image. The modified image is outputted.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to video processing, and moreparticularly, to a method of modifying a feature region of an imageaccording to an audio signal.

2. Description of the Prior Art

Web cameras are devices that typically include an image capturing devicewith a good refresh rate, and optionally a microphone for recordingsound in the form of voice or ambient noise. The web camera is usuallyconnected to a computing device, such as a personal computer or notebookcomputer, through a data interface, such as USB, or integrated with thecomputing device, e.g. in a housing of the notebook computer. The webcamera may be utilized as a video device by software for transmittingstreaming video and audio through a data network to provide videoconferencing and chat functions between two or more users in a chatsession.

As advanced video conferencing and chat technologies are developed, andas video chat grows in user base, users of video chat clients willdemand greater ability to customize the video stream sent to theirpeers. For example, as face detection technologies are refined, facialfeatures, such as hair, eyes, or skin may be modified, and themodifications may be made to track the location of the facial features.However, such modification is usually performed manually by the user,which can be cumbersome and inconvenient.

SUMMARY OF THE INVENTION

According to a first embodiment of the present invention, a method offacial image reproduction comprises retrieving an audio characteristicof an audio bitstream, receiving a video bitstream, extracting an imagefrom the video bitstream, extracting a feature region from the image,modifying the feature region according to the audio characteristic togenerate a modified image, and outputting the modified image.

According to the above embodiment of the present invention, anelectronic device for performing facial image reproduction comprises anaudio segmenting module for dividing the audio bitstream into aplurality of audio segments, a video segmenting module for dividing thevideo bitstream into a plurality of video segments, an audio processingmodule for retrieving an audio characteristic of the audio segments, animage extraction module for extracting an image from the video segments,a feature region detection module for extracting a feature region fromthe image, and an image modifying module for modifying the featureregion according to the audio characteristic to generate a modifiedimage.

According to a second embodiment of the present invention, a method ofmodifying an image based on an audio signal comprises capturing theimage, recording a sound, performing image analysis on the image,retrieving an audio characteristic from the recorded sound, andmodifying the image according to the audio characteristic to form amodified image.

According to the second embodiment of the present invention, acommunication system comprises a transmitting computing device and areceiving computing device. The transmitting computing device comprisesa system I/O interface for receiving an audio signal and a video signal,a processor for determining an audio characteristic of the sound,modifying an image of the video signal to generate the modified imageaccording to the audio characteristic and encoding the modified image,and a network interface for sending the encoded signal. The secondcomputing device comprises a network interface for receiving the encodedsignal from the transmitting computing device, a processor for decodingthe encoded signal to retrieve the modified image, and a displayinterface for outputting the modified image.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a method for performing facial image reproductionaccording to an embodiment of the present invention.

FIGS. 2A-2D are diagrams of an electronic device for realizing thefacial image reproduction method of FIG. 1.

FIG. 3 is a mapping table representing relationships between audiocharacteristics and modifications of feature regions.

FIG. 4 is a diagram of a communications system for realizing the facialimage reproduction method of FIG. 1.

FIG. 5 is a diagram of a method for performing facial image reproductionaccording to another embodiment of the present invention.

DETAILED DESCRIPTION

Please refer to FIG. 1, which is a diagram of a method of performingfacial image reproduction according to an embodiment of the presentinvention.

First, a video bitstream containing a facial image is received (Step100). The bitstream is a time series of bits for the transmission ofdigital data. The data transmission may be by cable connection,networking, telecommunications, etc. The video bitstream may be providedby an image capturing device.

An audio characteristic is retrieved from an audio bitstream (Step 102).This may be accomplished by receiving the audio bitstream and generatingthe audio characteristic by analyzing the audio bitstream. For example,an average volume may be calculated within a time period. Then, it canbe determined whether the average volume exceeds a threshold. Dependingon a result of determining whether the average volume exceeds thethreshold, a signal may be generated correspondingly. The audiocharacteristic retrieved in Step 102 may be retrieved by a frequencyanalysis, a rhythm detection, and/or by a tempo analysis. The audiobitstream itself may be encoded from music or speech. For speech, theaudio characteristic may be retrieved by analyzing tone of the speech orby speech recognizing technique.

A feature region is extracted from the video bitstream (Step 104). Thefeature region may be extracted from the video bitstream by extractingan image from the video bitstream, detecting a head region of the image,and extracting the feature region of the head region. The feature regionmay be extracted from the head region according to color information ofthe image, texture information of the image, and/or edge information ofthe image. Typical edge detection can be achieved by applying a Sobelfilter both horizontally and vertically. As for texture recognition,differences with neighboring pixels may be computed for each pixelwithin a certain region, and the differences may be summarized in ahistogram. Note that the texture information can be reflected by apattern of the histogram.

The feature region may be modified according to the audio characteristicto generate a modified image (Step 106). Modification of the featureregion may be accomplished in a number of different ways, includingmodifying coloration of the feature region according to the audiocharacteristic, and modifying texture of the feature region according tothe audio characteristic. For example, if the audio characteristic is avery high volume, one type of modification may be made, whereas a verylow volume may cause another type of modification.

The modified image may then be outputted through an output of theelectronic device (Step 108). And, in addition to outputting themodified image, the embodiment of the present invention may also outputthe audio bitstream. For example, the video bitstream may be dividedinto a plurality of video segments for efficiency in storage or forefficiency in further processing. Then, the modified image may beembedded in at least one of the video segments. Similarly, the audiobitstream may be divided into a plurality of audio segments. Then, theaudio segments may be synchronized with the modified video segments, andthe synchronized modified video segments and audio segments may beoutputted.

Please refer to FIGS. 2 a-2 d, which are diagrams of an electronicdevice for realizing the facial image reproduction method of FIG. 1.FIG. 2 a is a diagram of system architecture of the electronic device.The electronic device comprises a processor 270, memory 210, a displayinterface 230, data storage 240, a system I/O interface 250, and anetwork interface 260. The memory 210 stores program code 200 and anoperating system O/S. The display interface 230 controls a display 21,and the system I/O interface 250 is utilized for interfacing with anaudio capture device 22, an image capture device 23, and an audioreproducing device 24. The processor 270, the memory 210, the displayinterface 230, the data storage 240, the system I/O interface 250, andthe network interface 260 communicate with each other over a localinterface 220.

FIG. 2 b shows cooperation of modules of the program code 200 shown inFIG. 2 a. Each of the modules described below may be implemented insoftware code, but may also be implemented as an ASIC, or an FPGA. Theprogram code 200 comprises an audio segmenting module 201, a videosegmenting module 202, an audio characteristic extractor 280, an imageprocessing module 290, and a synchronizer 205. The audio segmentingmodule 201 receives the audio bitstream, e.g. music playing in theelectronic device or speech captured by a microphone, and segments theaudio bitstream to form an audio segment. The audio characteristicextractor 280 generates the audio characteristic corresponding to theaudio segment. The video segmenting module 202 receives the videobitstream, possibly from the image capture device 23, and segments thevideo bitstream to form a video segment. The image processing module 290modifies the video segment according to the audio characteristic. Thesynchronizer 205 synchronizes and encodes the audio segment and themodified video segment for output.

FIG. 2 c a diagram of the audio characteristic extractor 280 of FIG. 2b. The audio characteristic extractor 280 comprises an audiocharacteristic generator 287, which takes inputs from a volume analyzer281, a frequency analyzer 282, a rhythm analyzer 283, a tempo analyzer284, a tone analyzer 285, and a keyword recognizer 286. The volumeanalyzer 281 may analyze volume of the audio bitstream to generate avolume analysis result. The frequency analyzer 282 may analyze frequencyof the audio bitstream to generate a frequency analysis result. Therhythm analyzer 283 may analyze rhythm of the audio bitstream togenerate a rhythm analysis result. The tempo analyzer 284 may analyzetempo of the audio bitstream to generate a tempo analysis result. Thetone analyzer 285 may analyze tone of the audio bitstream to generate atone analysis result. The keyword analyzer 286 may analyze speech of theaudio bitstream to generate a keyword result. The audio characteristicgenerator 287 may then generate the audio characteristic based on all orpart of the above mentioned results. As an example, the volume analyzer281 may store volume data of the audio bitstream corresponding to a timeperiod in a storage module, and a signal generator may generate a signalcorresponding to a result of determining whether an average volume ofthe volume data exceeds a threshold. The signal then represents thevolume analysis result mentioned above that is generated by the volumeanalyzer 281.

FIG. 2D is a diagram of the image processing module 290 of FIG. 2B. Theimage processing module 290 comprises an image extraction module 291,which receives the video segment from the video segmenting module 202. Afeature region detection module 292 receives an image extracted by theimage extraction module 291, and detects a feature region of the image,such as the head region, the eyes, ears, hair, lips, etc. Detection maybe performed by analyzing color, texture, and/or edges of the image asdescribed above. A modifying module 293 modifies the feature region ofthe image according to the audio characteristic by looking up the audiocharacteristic in a database 295, which stores behavior corresponding tothe audio characteristic. An embedding module 294 then embeds themodified image into the video segment and outputs the video segment withthe modified image.

For the modifying module 293 to determine how to modify the image, theimage processing module 290 may further comprise the database 295, whichstores the behaviors corresponding to the audio characteristic. FIG. 3illustrates a mapping table that may be stored in the database 295,which represents relationships between the audio characteristic and thebehaviors that may be stored in the database 295. Depending on differentanalysis results, shown under “Detected Pattern of the Sound,”corresponding modifications may be made to the image, shown under“Corresponding Behavior of the Visual Effect.” For example, if a sharpincreasing in base frequency of the voice within a specific time isdetected by the rhythm analyzer 283, an effect may be applied withrespect to the rhythm, e.g. a face detected by the feature regiondetection module 292 may “blush” with the rhythm. In general, the vocalrange of human voice lies within a range of 100 Hz and 1000 Hz. The mostpopular basic frequency within this range may be detected by analyzinghuman voices. It is noted that when a person feels happy or shy, his orher voice may exhibit an increase in the basic frequency. Therefore, bydetecting that a “most popular basic frequency” shifts to a higher band,it may be determined that the human emotion changes and a specificeffect may be applied to the feature region. If a beat is detected bythe audio characteristic extractor 280, a blink effect may be applied intime with the beat. If a harmonic frequency response is detected by thefrequency analyzer 282, musical notes may be displayed around thefeature region. For speech, if a user's voice has an amplitude exceedinga predetermined threshold Thr1, and an average period between twoneighboring beats is below a predetermined threshold Thr2, saturation ofthe face may be enhanced, and blended with red. Finally, a snoringsymbol and a decreased eye size may be applied to the feature regionwhen the amplitude of the voice is below a threshold Thr3, indicatingthat the user is silent. Of course, many other modifications arepossible. For example, an excited setting, a quiet setting, or othertypes of settings may be set based on statistical models related to thevolume, frequency, and/or rhythm of the sound. Types of modificationsthat may be made include replacements of pixels corresponding to thefeature regions, or modifications of pixels corresponding to the featureregions. For example, the hair of the user may be replaced with apredetermined hair image. Or, the skin may be tinted in a gradientmanner according to different levels of the volume, the frequency, orthe rhythm. All modifications may be predetermined by default, or may becustomized by the user. For example, the user may record an image orvideo clip for use in modifying the image according to the predeterminedsetting.

FIG. 4 is a diagram of a communications system for realizing the facialimage reproduction method of FIG. 1. As shown in FIG. 4, a first(transmitting) computing device 40A and a second (receiving) computingdevice 40B may communicate through a network. The transmitting computingdevice 40A may be a transmitting personal computer (PC), and thereceiving computing device 40B may be a receiving personal computer(PC). The personal computers 40A, 40B respectively comprise a processor470A, 470B, a memory 410A, 410B, a local interface 420A, 420B, a displayinterface 430A, 430B, data storage 440A, 440B, a system I/O interface450A, 450B, and a network interface 460A, 460B. Each personal computer40A, 40B may interface with a display 41A, 41B, an audio capture device42A, 42B, such as a microphone, an image capture device 43A, 43B, andthe network. Of course, the audio capture device 42A, 42B and the imagecapture device 43A, 43B may be integrated, e.g. in a webcam. In theembodiment shown in FIG. 4, the memory 410A, 410B stores an operatingsystem and program code 200A, 200B similar to the program code 200described above. The processor 470A, 470B, the memory 410A, 410B, thedisplay interface 430A, 430B, the data storage 440A, 440B, the systemI/O 450A, 450B, and the network interface 460A, 460B may communicatewith each other through the local interface 420A, 420B.

The communication system shown in FIG. 4 may be utilized to providetwo-way video chat over the network between a user of the transmittingPC 40A and a user of the receiving PC 40B. Considering only the first PC40A, a chat client may be displayed in the display 41A, and video of theusers may be captured by the image capture devices 43A, 43B for displayin the display 41A. The video of the users may be modified to addeffects dependent on audio.

Referring to FIG. 5, a process 50 may be utilized in the communicationssystem to modify the video captured by the image capture device 42A orthe image capture device 42B. First, an image is captured (Step 500),and a sound is recorded (Step 501). Image analysis is performed on theimage (Step 502), and an audio characteristic is retrieved from therecorded sound (Step 503). Then, the image may be modified according tothe audio characteristic to form a modified image (Step 504), themodified image may be encoded with the sound to form an encoded modifiedsignal (Step 505), and the encoded signal may be transmitted (Step 506).It is also acceptable for the image and sound to be encoded first toform an encoded unmodified signal (Step 507), and for the encodedmodified signal to be transmitted with the audio characteristic (Step508). In this way, a receiver of the encoded unmodified signal and theaudio characteristic, e.g. the second PC 40B, may modify the encodedunmodified signal according to the audio characteristic prior toplayback. In other words, modification may be performed prior toencoding or post encoding, and may be performed by either thetransmitting device or the receiving device.

Thus, the method, electronic device, and communication system accordingto embodiments of the present invention allow the user to enhancehis/her video stream conveniently based on sounds being processed. Thisenhances the video chat experience, allows for greater interactionbetween users, and also provides entertainment for all users in the chatsession.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention.

What is claimed is:
 1. A communication system comprising: a transmittingcomputing device comprising: a system I/O interface for receiving anaudio signal and a video signal; a processor for determining an audiocharacteristic of the sound, extracting a facial feature region from animage of the video signal, modifying the extracted facial feature regionof the image of the video signal to express human emotion changesindicated by the audio characteristic of the sound to generate themodified image and encoding the modified image; and a network interfacefor sending the encoded signal; and a receiving computing devicecomprising: a network interface for receiving the encoded signal fromthe transmitting computing device; a processor for decoding the encodedsignal to retrieve the modified image; and a display interface foroutputting the modified image.